NATURAL LANGUAGE GENERATION TASKS

Natural Answer Generation with Heterogeneous Memory (N18-1)

Challenge:	This tutorial aims to bring awareness of the important and emerging research area of open-domain creative generation.
Approach:	They will review recent studies on creative language generation at sentence level as well as longer forms of text.
Outcome:	This paper reviews recent studies on creative language generation at sentence level as well as longer forms of text.

Copied to clipboard

Yao Fu, Yansong Feng

PROM: Pivoted and Regulated Optimization for Multilingual Instruction Learning (2025.naacl-short)

Challenge:	Recent work on memory augmented encoder-decoder frameworks has shown promising progress for natural language generation tasks.
Approach:	They propose a memory-augmented encoder-decoder framework that takes care of memory contents from different sources to explicitly avoid repetition.
Outcome:	The proposed approach can produce readable and meaningful answer sentences while maintaining high coverage for given answer information.

Copied to clipboard

Jaeseong Lee, Seung-won Hwang, Hojin Lee, Yunju Bak, Changmin Lee

Multilinguality

Challenge:	Existing solutions to large language models (LLMs) are English-centric, hindering their application to 6500+ existing languages.
Approach:	They propose to append English tuning data with its translated pair to solve this problem . they identify English as an internal pivot language and propose to regulate between them .
Outcome:	The proposed model is able to generalize on multiple benchmarks across different languages.

Naver Labs Europe’s Systems for the Document-Level Generation and Translation Task at WNGT 2019 (D19-56)

Copied to clipboard

Fahimeh Saleh, Alexandre Berard, Ioan Calapodescu, Laurent Besacier

AMRFact: Enhancing Summarization Factuality Evaluation with AMR-Driven Negative Samples Generation (2024.naacl-long)

Challenge:	Recent advances in machine translation and natural language generation have created many challenges in this field especially when context is considered.
Approach:	They propose to leverage data from machine translation and natural language generation tasks to do transfer learning between MT, NLG and MT with source-side metadata.
Outcome:	The proposed approach outperforms the previous state-of-the-art on the Rotowire NLG task.

Copied to clipboard

Haoyi Qiu, Kung-Hsiang Huang, Jingnong Qu, Nanyun Peng

Summarization

Challenge:	Existing methods for evaluating factual consistency of abstractive summarization lack coherence or error-type coverage.
Approach:	They propose a framework that generates perturbed summaries using Abstract Meaning Representations (AMRs) they use a selection module NegFilter to ensure the quality of the generated negative examples .
Outcome:	The proposed framework outperforms existing systems on the AggreFact-SOTA benchmark and provides high error-type coverage.

Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting (2021.emnlp-main)

Copied to clipboard

Wangchunshu Zhou, Tao Ge, Canwen Xu, Ke Xu, Furu Wei

Fidelity-Enriched Contrastive Search: Reconciling the Faithfulness-Diversity Trade-Off in Text Generation (2023.emnlp-main)

Challenge:	Existing text infilling objectives for pretrained language models require self-supervision by masking out tokens or spans in text.
Approach:	They propose to extend text infilling to a self-supervised sequence-to-sequence (Seq2Sequen) task.
Outcome:	The proposed task improves the model's performance on various natural language generation tasks.

Copied to clipboard

Wei-Lin Chen, Cheng-Kuang Wu, Hsin-Hsi Chen, Chung-Chi Chen

On Decoding Strategies for Neural Text Generators (2022.tacl-1)

Challenge:	Language models often generate fluent and convincing content but can lack consistency with the provided source, resulting in potential inaccuracies.
Approach:	They propose a new decoding method that augments the contrastive search framework with context-aware regularization terms to promote tokens that are semantically similar to the provided source while penalizing repetitiveness in the generated text.
Outcome:	The proposed method improves faithfulness across various language models while maintaining output diversity comparable to well-performing decoding algorithms.

Copied to clipboard

Gian Wiher, Clara Meister, Ryan Cotterell

FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Large Language Models (2025.acl-long)

Challenge:	a recent study suggests that decoding strategies may be more important than the model architecture itself when generating text from probabilistic models.
Approach:	They propose to measure changes in attributes of generated text as a function of decoding strategy and task using human and automatic evaluation.
Outcome:	The proposed study shows that decoding strategies do not always transfer across tasks . authors show that the differences in attributes are not always consistent across tasks, they say .

Copied to clipboard

Raghav Singhal, Kaustubh Ponkshe, Praneeth Vepakomma

Explanation Graph Generation via Pre-trained Language Models: An Empirical Study with Contrastive Learning (2022.acl-long)

Challenge:	Existing methods for low-rank averaging of LoRA adapters result in inexact updates.
Approach:	They propose a method which adds a residual error term to the pre-trained frozen weight matrix to achieve exact updates with minimal computational and communication overhead.
Outcome:	The proposed method achieves exact updates with minimal computational and communication overhead, preserving LoRA’s efficiency.

Copied to clipboard

Swarnadeep Saha, Prateek Yadav, Mohit Bansal

An Empirical Study of Building a Strong Baseline for Constituency Parsing (P18-2)

Challenge:	Pre-trained sequence-to-sequence language models generate structured outputs such as graphs with limited supervision.
Approach:	They propose to use pre-trained sequence-to-sequence language models to generate graphs . they propose to learn structural constraints and semantics of graphs with limited supervision .
Outcome:	The proposed models can learn structural constraints and semantics of graphs with limited supervision.

Copied to clipboard

Jun Suzuki, Sho Takase, Hidetaka Kamigaito, Makoto Morishita, Masaaki Nagata

Small Language Models Improve Giants by Rewriting Their Outputs (2024.eacl-long)

Challenge:	Sequence-to-sequence models have been used for natural language generation tasks such as machine translation and summarization.
Approach:	They propose to build a strong baseline based on general purpose sequence-to-sequence models for constituency parsing.
Outcome:	The proposed model outperforms existing models in natural language generation tasks without any explicit task-specific knowledge or architecture of constituent parsing.

Copied to clipboard

Giorgos Vernikos, Arthur Brazinskas, Jakub Adamek, Jonathan Mallinson, Aliaksei Severyn, Eric Malmi

GRAMMAR-LLM: Grammar-Constrained Natural Language Generation (2025.findings-acl)

Challenge:	despite impressive performance of large language models, they lag behind specialized models in various tasks.
Approach:	They propose a training model that can be integrated with different LLMs at inference to improve their performance without task-specific training.
Outcome:	The proposed model outperforms standard models on four natural language generation tasks.

Copied to clipboard

Gabriele Tuccio, Luana Bulla, Maria Madonia, Aldo Gangemi, Misael Mongiovì

Likelihood-based Mitigation of Evaluation Bias in Large Language Models (2024.findings-acl)

Challenge:	Existing approaches to fine-tuning and prompting are insufficient to ensure compliance with predefined taxonomies, syntactic structures, or domain-specific rules.
Approach:	They propose a framework that integrates formal grammatical constraints into the decoding process to enforce syntactic correctness in linear time while maintaining expressiveness in grammar rule definition.
Outcome:	The proposed framework enforces syntactic correctness in linear time while maintaining expressiveness in grammar rule definition.

Copied to clipboard

Masanari Ohi, Masahiro Kaneko, Ryuto Koike, Mengsay Loem, Naoaki Okazaki

The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text (2024.findings-naacl)

Challenge:	Large Language Models (LLMs) are widely used to evaluate natural language generation tasks as automated metrics.
Approach:	They propose to use LLMs to evaluate sentences with higher likelihoods and lower likelihoods to mitigate the likelihood bias.
Outcome:	The proposed method overrates sentences with higher likelihoods while underrating sentences with lower likelihoods.

Copied to clipboard

Yanzhu Guo, Guokan Shang, Michalis Vazirgiannis, Chloé Clavel

On Hallucination and Predictive Uncertainty in Conditional Language Generation (2021.eacl-main)

Challenge:	a new study examines the effects of training language models on synthetic data generated by their predecessors.
Approach:	They propose to use recursive finetuning techniques to assess linguistic diversity of models.
Outcome:	The proposed metrics show a decrease in diversity of model outputs through successive iterations, especially for tasks demanding high levels of creativity.

Copied to clipboard

Yijun Xiao, William Yang Wang

KFCNet: Knowledge Filtering and Contrastive Learning for Generative Commonsense Reasoning (2021.findings-emnlp)

Challenge:	Modern deep neural network models have brought drastic improvements in generation quality measured by standard metrics on different natural language generation tasks.
Approach:	They propose a beam search extension to reduce hallucination in conditional language generation by adding a prediction extension to beam search.
Outcome:	The proposed extension improves trading performance on standard metric for less hallucination with the proposed beam search variant.

Copied to clipboard

Haonan Li, Yeyun Gong, Jian Jiao, Ruofei Zhang, Timothy Baldwin, Nan Duan

Let’s do it “again”: A First Computational Approach to Detecting Adverbial Presupposition Triggers (P18-1)

Challenge:	Pre-trained language models have led to substantial gains over a broad range of NLP tasks, but have limitations for high-quality tasks such as commonsense generation and ad keyword generation.
Approach:	They propose a Knowledge Filtering and Contrastive learning Network which references external knowledge and achieves better generation performance.
Outcome:	The proposed model outperforms the current state of the art on the CommonGen benchmark by a large margin.

Copied to clipboard

Andre Cianflone, Yulan Feng, Jad Kabbara, Jackie Chi Kit Cheung

Challenge:	a novel task of predicting adverbial presupposition triggers is useful for natural language generation . a focus is on a new attention mechanism for predicting presuposition trigger .
Approach:	They propose a new attention mechanism for predicting adverbial presupposition triggers . they propose to augment a baseline neural network without additional trainable parameters .
Outcome:	The proposed model outperforms baseline models in predicting adverbial presupposition triggers.

Rethinking the Agreement in Human Evaluation Tasks (C18-1)

Copied to clipboard

Jacopo Amidei, Paul Piwek, Alistair Willis

DeltaScore: Fine-Grained Story Evaluation with Perturbations (2023.findings-emnlp)

Challenge:	In natural language processing, IAA is often viewed as a means of assessing the quality of data on a task, in particular, the reliability.
Approach:	They propose a new approach to use agreement metrics in natural language generation evaluation tasks to reduce subjective bias.
Outcome:	The proposed approach is based on the inter-annotator agreement (IAA) of natural language generation tasks.

Copied to clipboard

Zhuohan Xie, Miao Li, Trevor Cohn, Jey Lau

Challenge:	Existing evaluation metrics for stories are limited in assessing intricate aspects of storytelling, such as fluency and interestingness.
Approach:	They propose a novel method that uses perturbation techniques to evaluate story aspects . they compare fluency, coherence, relatedness, logicality, interestingness and interestingness to existing metrics .
Outcome:	The proposed method shows that one specific perturbation is highly effective in capturing multiple aspects.

Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021.acl-long)

Copied to clipboard

Xiang Lisa Li, Percy Liang

Solving Aspect Category Sentiment Analysis as a Text Generation Task (2021.emnlp-main)

Challenge:	Fine-tuning is the prevalent paradigm for using large pretrained language models for downstream tasks, but it requires updating and storing all the parameters of the LM.
Approach:	They propose a lightweight alternative to fine-tuning for natural language generation tasks that optimizes a sequence of continuous vectors, which they call the prefix.
Outcome:	The proposed approach outperforms fine-tuning in the full data setting and extrapolates better to examples with topics that are unseen during training.

Copied to clipboard

Jian Liu, Zhiyang Teng, Leyang Cui, Hanmeng Liu, Yue Zhang

Positional Encoding to Control Output Sequence Length (N19-1)

Challenge:	Existing methods for Aspect category sentiment analysis use pre-trained language models to learn aspect category-specific representations.
Approach:	They propose to make use of pre-trained language models by casting the ACSA tasks into natural language generation tasks, using natural language sentences to represent the output.
Outcome:	The proposed method gives the best reported results, having large advantages in few-shot and zero-shot settings.

Copied to clipboard

Sho Takase, Naoaki Okazaki

Keeping Notes: Conditional Natural Language Generation with a Scratchpad Encoder (P19-1)

Challenge:	Neural encoder-decoder models have been successful in natural language generation tasks, but they must be limited to a specified length for abstractive summarization.
Approach:	They propose a sinusoidal positional extension to preserve the length constraint so that a neural encoder-decoder model can generate a text of any length even if the target length is unseen in training data.
Outcome:	The proposed method can generate a text of any length even if the target length is unseen in training data and improves ROUGE scores.

Copied to clipboard

Ryan Benmalek, Madian Khabsa, Suma Desu, Claire Cardie, Michele Banko

Generalized Supervised Attention for Text Generation (2021.findings-acl)

Challenge:	Qualitative assessments in the form of human judgements (question generation), attention visualization (MT), and sample output (summarization) provide further evidence of the ability of Scratchpad to generate fluent and expressive output.
Approach:	They propose to use the encoder as a "scratchpad" memory to keep track of what has been generated and guide future generation.
Outcome:	The proposed mechanism improves the fluency of seq2seq models on three well-studied natural language generation tasks.

Copied to clipboard

Yixian Liu, Liwen Zhang, Xinyu Zhang, Yong Jiang, Yue Zhang, Kewei Tu

SGG: Learning to Select, Guide, and Generate for Keyphrase Generation (2021.naacl-main)

Challenge:	Existing supervised attention methods that use human knowledge to learn better alignments are costly or infeasible.
Approach:	They propose a generalized supervised attention method based on quasi alignments that are easier to obtain than ideal alignments.
Outcome:	The proposed framework improves generation performance and is robust against errors in attention supervision.

Copied to clipboard

Jing Zhao, Junwei Bao, Yifan Wang, Youzheng Wu, Xiaodong He, Bowen Zhou

Autoregressive Knowledge Distillation through Imitation Learning (2020.emnlp-main)

Challenge:	Existing keyphrase generation approaches synchronously generate present and absent keyphrases without explicitly distinguishing these two categories.
Approach:	They propose to deal with present and absent keyphrases separately with different mechanisms by using a hierarchical neural network with a pointing-based selector and a selection-guided generator.
Outcome:	The proposed model outperforms baselines on four keyphrase generation tasks and shows extensibility in natural language generation tasks.

Copied to clipboard

Alexander Lin, Jeremy Wohlwend, Howard Chen, Tao Lei

Creative Preference Optimization (2025.findings-emnlp)

Challenge:	Autoregressive models are ubiquitous in natural language processing due to the sequential nature of text generation.
Approach:	They propose a compression technique for autoregressive models driven by an imitation learning perspective on knowledge distillation.
Outcome:	The proposed method outperforms other distillation algorithms on translation and summarization tasks while increasing inference speed 14 times.

Copied to clipboard

Mete Ismayilzada, Antonio Laverghetta Jr., Simone A. Luchini, Reet Patel, Antoine Bosselut, Lonneke Van Der Plas, Roger E. Beaty

Challenge:	Existing methods for enhancing LLM creativity focus on diversity or specific tasks, failing to address creativity’s multifaceted nature in a generalizable way.
Approach:	They propose a method that injects signals from multiple creativity dimensions into the preference optimization objective in a modular fashion.
Outcome:	The proposed method outperforms baseline models on automated and human evaluations while maintaining high output quality.

Benchmarking Large Language Model Capabilities for Conditional Generation (2023.acl-long)

Copied to clipboard

Joshua Maynez, Priyanka Agrawal, Sebastian Gehrmann

Contextualized Sequence Likelihood: Enhanced Confidence Scores for Natural Language Generation (2024.emnlp-main)

Challenge:	Autoregressive and pre-trained large language models have shifted the field from application-specific to generation-based approaches.
Approach:	They propose to adapt existing application-specific generation benchmarks to pre-trained large language models to better suit different tasks.
Outcome:	The proposed models differ in their applicability to different data regimes and their generalization to multiple languages.

Copied to clipboard

Zhen Lin, Shubhendu Trivedi, Jimeng Sun

Natural Language Interfaces

Challenge:	Currently, the most commonly used confidence score is the likelihood of the generated sequence . different tokens should be weighted differently depending on the context.
Approach:	They propose to assign different weights to various tokens using attention values elicited from the base LLM.
Outcome:	The proposed model improves the confidence of the predicted sequence probability by assigning weights to tokens based on attention values elicited from the base model.

Set to Ordered Text: Generating Discharge Instructions from Medical Billing Codes (D19-1)

Copied to clipboard

Litton J Kurisinkel, Nancy Chen

In-sample Curriculum Learning by Sequence Completion for Natural Language Generation (2023.acl-long)

Challenge:	a neural architecture learns to generate content in a specific order without explicit specifications of the relations between input entities and output entities.
Approach:	They propose a natural language generation task that generates discharge instructions from ICD codes . they propose to model content ordering and text generation in a specific order .
Outcome:	The proposed model outperforms baseline models in BLEU scores and human evaluation.

Copied to clipboard

Qi Jia, Yizhu Liu, Haifeng Tang, Kenny Zhu

Select, Prompt, Filter: Distilling Large Language Models for Summarizing Conversations (2023.emnlp-main)

Challenge:	Existing work on curriculum learning rely on task-specific expertise and cannot generalize to different tasks.
Approach:	They propose to do in-sample curriculum learning for natural language generation tasks using human-crafted rules and a numeric score for each sample based on domain expertise to rank the model.
Outcome:	The proposed learning strategy generalizes well to different tasks and achieves significant improvements over baselines.

Copied to clipboard

Minh-Quang Pham, Sathish Indurthi, Shamil Chollampatt, Marco Turchi

Why Does Zero-Shot Cross-Lingual Generation Fail? An Explanation and a Solution (2023.findings-acl)

Challenge:	Large language models (LLMs) can be expensive to train, deploy, and use for specific natural language generation tasks.
Approach:	They propose a method to distill ChatGPT and fine-tune smaller LMs for summarizing forum conversations using a semantic similarity metric.
Outcome:	The proposed method leads to significant improvements of up to 6.6 ROUGE-2 score by leveraging sufficient in-domain pseudo-labeled data over standard KD approach given the same size of training data.

Copied to clipboard

Tianjian Li, Kenton Murray

Responsible & Trustworthy NLP

Challenge:	Existing studies on cross-lingual transferability of multilingual LMs show that they can perform tasks in low-resource languages.
Approach:	They propose a method to regularize the model from learning language invariant representations and a way to select model checkpoints without a development set in the target language.
Outcome:	The proposed method reduces the accidental translation problem by 68% and improves the ROUGE-L score by 1.5 on average.

On Improving Summarization Factual Consistency from Natural Language Feedback (2023.acl-long)

Copied to clipboard

Yixin Liu, Budhaditya Deb, Milagro Teruel, Aaron Halfaker, Dragomir Radev, Ahmed Hassan Awadallah